Effects of Tunable Data Compression on Geophysical Products 
Retrieved from Surface Radar Observations with Applications to 
Spaceborne Meteorological Radars 


Philip M. Gabriel 

Colorado State University, Fort Collins, Colorado, USA 


PENSHU yeh and Si-Chee tsay 
NASA Goddard Space Flight Center, Greenbelt, Maryland, USA 


Submitted to Journal of Atmospheric and Oceanic Technology on 


December 9, 2013 

Keywords: Radar observation; Data compression; Cloud properties; CCSDS 


Corresponding Author: 

Dr. Philip M. Gabriel 

Colorado State University 

Fort Collins, CO 80523 

Tel: +1-970-491-8670 

Fax: +1-970-491-8449 

E-mail address: gabriel@atmos.colostate.edu 



Abstract 


1 
2 

3 This paper presents results and analyses of applying an international space data 

4 compression standard to weather radar measurements that can easily span 8 orders of 

5 magnitude and typically require a large storage capacity as well as significant bandwidth 

6 for transmission. By varying the degree of the data compression, we analyzed the non- 

7 linear response of models that relate measured radar reflectivity and/or Doppler spectra to 

8 the moments and properties of the particle size distribution characterizing clouds and pre- 

9 cipitation. Preliminary results for the meteorologically important phenomena of clouds 

10 and light rain indicate that for a ±0.5 dB calibration uncertainty, typical for the ground- 

11 based pulsed-Doppler 94 GHz (or 3.2 mm, W-band) weather radar used as a proxy for 

12 spacebome radar in this study, a lossless compression ratio of only 1.2 is achievable. 

13 However, further analyses of the non-linear response of various models of rainfall rate, 

14 liquid water content and median volume diameter show that a lossy data compression ra- 

15 tio exceeding 15 is realizable. The exploratory analyses presented are relevant to future 

16 satellite missions, where the transmission bandwidth is premium and storage require- 

17 ments of vast volumes of data, potentially problematic. 

18 


i 



19 1 . Introduction 

20 

21 Observations of atmospheric processes for the purpose of understanding, diagnos- 

22 ing, predicting and projecting weather and climate rely increasingly on the analysis of 

23 data from a host of instruments that include surface-based, suborbital and spaceborne ra- 

24 dars, lidars as well as imaging spectrometers. Undoubtedly, employment of suites of in- 

25 struments on either space/airborne or ground platforms will generate vast volumes of data 

26 that can quickly overwhelm data storage and transmission bandwidths. To alleviate data 

27 congestion, various approaches to data processing, editing and compression techniques 

28 have been studied. However, the most relevant question is “ if and how does the pro- 

29 cessing technique affect the end products used in understanding and predicting weather 

30 and climate ?” To address this question, we will first investigate the effects of data com- 

31 pression, using the Consultative Committee for Space Data Systems (CCSDS, 2005) 

32 “ Image Data Compression ” standard on ground-based, (inherently noisy) weather radar 

33 signals. Studies connected to the applications of this standard to spectroscopic observa- 

34 tions (which span a much smaller dynamic range) have been performed (e.g., Barrie et 

35 al., 2009; Garcia- Vilchez and Serra-Sagrista, 2009). However, to the best of the authors’ 

36 knowledge, studies characterizing the effects of the CCSDS data compression algorithm 

37 to radar data and its derived products have not been conducted. As such, the results pre- 

38 sented here are timely in that they demonstrate the achievable onboard compression for 

39 selected applications while underscoring the benefits of such analyses. Our characteriza- 

40 tion will provide crucial information for current (e.g., Earth Observing System, 1999) and 

41 future missions (e.g., Decadal Survey and Venture Class missions in NASA Strategic 

42 Plan, 2011). 
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43 

44 The space data compression standard algorithm used in this study was derived by 

45 the CCSDS body composed of major international space agencies with NASA as a major 

46 partner ( www.ccsds.org) . Commonly known compression techniques generally fall into 

47 either the fully lossless, or the lossy categories (Sayood, 2012). The lossless technique 

48 preserves data fidelity with very limited data reduction performance while the lossy tech- 

49 niques with good perfonnance require much sophisticated computation as in JPEG2000 

50 (Taubman el al., 2002). The CCSDS standard addresses space implementation constraints 

51 such as power, computation resources and a relatively high required throughput with ex- 

52 cellent performance. Additionally it provides user precisely selectable data reduction ra- 

53 tio from highly lossy to full lossless, i.e., tunable. This feature allows flexibility in space- 

54 craft downlink rate allocation amongst multiple science instruments. The former guaran- 

55 tees the restored data identical to the original; the latter generally furnishes higher com- 

56 pression ratios but introduces some level of distortion in the reconstructed data. This al- 

57 gorithm allows a user to directly control the compressed data volume or the fidelity with 

58 which the data can be reconstructed. The higher fidelity required by lossless compression 

59 results in a higher volume of compressed data for a given source data set. The compres- 

60 sion ratio (CR) is defined as the ratio of the number of bits per sample before compres- 

61 sion to that of the encoded data. With larger CR, the total data volume that needs to be 

62 transmitted is much reduced. For example, at CR=24, the volume is l/12th of the volume 

63 obtained at CR=2. A larger CR not only requires less onboard storage (if needed), it is 

64 less demanding in terms of either narrower bandwidth for transmission within a fixed 

65 time frame, or a much reduced transmission time period given a fixed bandwidth. How- 

66 ever, increasing the CR introduces increasing reconstruction noise in the decompressed 
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67 data. 

68 While lossless compression is mandatory for many types of data (e.g. complied com- 

69 puter codes), measurements with inherent noise need not be kept perfectly intact for 

70 transmission or storage provided the introduced distortions are below the inherent noise 

71 levels. The pressing needs for yielding higher compression ratios for certain types of ap- 

72 plications, formulated in terms of the previously posed question is the major drive for the 

73 current study. We contend that onboard data compression of spaceborne radar, lidar and 

74 spectroscopic observations of the Earth-atmosphere system must advance in lockstep and 

75 eventually unite in an indistinguishable fashion. We envision a future in which archives 

76 of these suites of instruments output will not be monstrous dumps of data, but rather, the 

77 information mined from these data, occupying a fraction of the volume and coded in a 

78 format that is more useful to the scientific communities and to policy makers. In the 

79 meantime, it is necessary to evaluate the existing lossy compression algorithm developed 

80 for use in spaceborne platforms, applied here, to radar observations. 

81 Because W-band radars differ in several respects from those operating at lower fre- 

82 quencies, we provide a brief background on their salient characteristics that are exploited 

83 in spaceborne observations of the Earth’s atmosphere. W-band pulsed-Doppler radars are 

84 employed since they exhibit great sensitivity arising from the proportionality of the 

85 backscattering cross-section in the Rayleigh regime (D « A,) to 1/A, 4 , where D is the parti- 

86 cle diameter and A,, the wavelength. Such radars are capable of detecting particles with 

87 diameters of tens of microns, typically found in clouds and in light precipitation. In addi- 

88 tion, they can be configured to have excellent temporal and spatial resolution and can op- 

89 erate with physically small antennas that have a very narrow beamwidth, resulting in 
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90 sampling volumes that are very small compared with those of longer wavelength radars. 

91 This reduced sampling volume decreases the effects of the Doppler spectrum broadening 

92 due to turbulence. These features of W-band radars, compounded with their portability 

93 and their ability to measure range-resolved velocities of particles, make them powerful 

94 tools for studying the macrophysics/microphysics of frequently occurring boundary-layer 

95 stratocumulus and widespread high-altitude cirrus clouds. 

96 According to Lhennitte (1988), the deep Mie backscattering oscillations occurring 

97 in the raindrop particle size range make W-band radars an attractive choice for vertical air 

98 motion and particle size distribution measurements, particularly when used in conjunc- 

99 tion with an S-band (e.g., 2-4 GHz) or an X-band (e.g., 8-12 GHz) radar. Furthermore, 

100 when W-band radars are used with longer wavelength radars, estimation of cloud liq- 

101 uid/ice water content in precipitating clouds is possible (e.g., Gaussiat et al., 2003). Al- 

102 ready, some of the stated advantages of W-band radar are being realized by the CloudSat 

103 mission (Stephens et al., 2002), even though in the radar employed, velocity measure- 

104 ment capability by the Doppler effect is absent. However, the spacebome W-band radar 

105 to be used in the upcoming, European-Japanese EarthCare mission (Bezy et al., 2005) 

106 will include Doppler processing. For the reasons just discussed, it is expected that future 

107 spaceborne observation platforms will incorporate multi-frequency radars (as well as li- 

108 dars and other passive instruments such as spectrometers); hence the critical need for ad- 

109 vanced data compression techniques. Before proceeding, we acknowledge that there are 

110 significant differences between surface and spaceborne radars. The latter move at a high 

111 velocity and consequently, smear the scene below. This work addresses only the effects 

112 of data compression and not effects attributed to motion of the radar platfonn. We con- 


4 



113 tend that as long as the complexity of the meteorological scenes is comparable, the results 

114 we obtain are transferable. 

115 To begin to address the crucial question posed earlier, the paper is divided as fol- 

116 lows: Section 2 describes the methodology. Here, a description is provided of the prepa- 

117 ration of the data to be compressed using the CCSDS standard. Also included is a brief 

118 overview of the compression standard. Data products depending non-linearly on the radar 

119 reflectivity are taken from the literature (e.g., the rainfall rate, liquid water concentration 

120 and median volume diameter) and the procedure used to evaluate the effect of the stand- 

121 ard is given. Attenuation by gaseous absorption and precipitation, as they impact the W- 

122 band radar are discussed. Section 3 shows the effects of data compression on the afore- 

123 mentioned products. Concluding remarks and future work are given in Section 4. 

124 

125 2. Methodology 

126 

127 2.1 Data Source 

128 

129 Pulsed-Doppler W-band radar signals, provided by SMARTLabs/ACHIEVE ( cf. 

130 http://smartlabs.gsfc.nasa.gov/) mobile laboratory pictured in Fig. la, were acquired us- 

131 ing a commercial receiver. The output from the receiver front-end (i.e., from the in-phase, 

132 /, and the quadrature components, Q) was digitized using a pair of 16 bits-sample' 1 analog 

133 to digital converters running at a data rate of 50xl0 6 samples-sec" 1 and converted to dou- 

134 ble-precision reflectivity data, whose minimum discernible value is -55 dBZ at 1 km. For 

135 this study, W-band radar reflectivity measurements of a complex weather system occur- 

136 ring over GSFC on 8 May 2012 were used to demonstrate the performance of lossless 

137 and lossy data compression. The test-bed data shown in Fig. lb were obtained when the 

138 W-band radar, running at a pulse repletion frequency (PRF) of 5482 Hz was zenith point- 
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139 ing with vertical resolution set to 24 meters in 524 range bins, for a maximum range of 

140 12.576 km. The total observation time of 1,800 seconds is comprised of 7,709 dwell time 

141 intervals, each interval spanning 0.233 seconds; hence, 4,039,516 points constitute the 

142 reflectivity image. Furthermore, as depicted in Fig. lb, a large dynamic range of reflec- 

143 tivity measurements was acquired within the duration of 30 minutes, starting at 

144 18:27:24UTC. Retrieved cloud products (e.g., cloud top temperature, height, etc.) in- 

145 ferred from the overpass of MODIS sensors onboard EOS/Aqua at 18:05 UTC, indicated 

146 the presence of a large multi-layer, multi-phase (ice/melting/liquid) cloud rain system. 

147 The corresponding W-band linear depolarization ratio (LDR), shown in Fig. lc can dif- 

148 ferentiate ice ( — 20dB), melting ( — lOdB, ice coated with water, peaking at -3.5 km 

149 range in Fig. lb) and water (- -30dB) cloud phases. The mean fall velocity shown in Fig. 

150 Id is also indicative of drizzle/rain reaching the radar site, occurring within an elapsed 

151 time of -7.5 minutes. 

152 Reflectivity is a measure of a radar target’s efficiency in intercepting and returning 

153 radiofrequency energy that depends upon the size, shape, orientation, and dielectric prop- 

154 erties of the target. In the meteorological context, reflectivity finds prolific use in infer- 

155 ring characteristics of clouds and precipitation that are fundamental, such as the particle 

156 size distribution of clouds, liquid/ice water content and rainfall/snowfall rates. While this 

157 multi-parameter radar is capable of displaying LDR and hydrometeor velocity profiles, 

158 attention here has been restricted to reflectivity data only, since such data exhibits the 

159 greatest dynamic range. It is expected that quantities characterized by a smaller dynamic 

160 range such as those shown in Figs, lc and Id can be compressed using larger compres- 

161 sion ratios. Hence, the motivation of this study is to understand how perturbations intro- 
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162 duced by lossy data compression affect derived products. 

163 2.2 Compression Technique 

164 

165 The CCSDS tunable Image Data Compression standard employs a 2-D discrete 

166 wavelet transform (DWT) to decompose input image into wavelet coefficients. These 

167 coefficients are then selected according to their energy levels through the use of a bit 

168 plane encoder (BPE), which codes them at each bit plane. With this algorithm, users can 

169 easily, after analyzing the raw image data, make decisions on the desirable compression 

170 ratio for the image under consideration. In fact, the reallocation of such desirable corn- 

171 pression ratio can be applied after the image has been compressed at a lower than desired 

172 compression ratio if pre-compression data analysis is un-suitable (e.g., due to unavailabil- 

173 ity of onboard processing power). The selected final higher compression ratio can be ap- 

174 plied by simply truncating the previously compressed bit stream because of the nature of 

175 the “ embedded bit stream ” property of the algorithm. Such property guarantees that the 

176 highly compressed image infonnation is located at the front part of the coded bit stream, 

177 followed by bit stream which improves the fidelity of the compression but lowers corn- 

178 pression ratio. Figure 2 describes how the compression standard can be applied to facili- 

179 tate optimal onboard resource utilization when data from multiple instruments have to be 

180 adjusted for downlink rate allocation. For use in the compression algorithm, the entire 

181 data set was first offset by 55 dB so that the smallest datum is 0 dBZ. A scaling factor, k, 

182 of 2 18 was chosen to reproduce the dynamic range spanned by the reflectivity. The choice 

183 of the number of bits (here 18) is determined by the intrinsic variance of the reflectivity. 

184 However, the latter is difficult to compute because: (1) the reflectivity is not strictly a 

185 function of PRF, since the sample values from pulse-to-pulse are not independent, (2) the 
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variance of the reflectivity is a complex function of the velocity spectral width of the 
cloud, (3) the variance depends on the PRF, the radar wavelength, the fast Fourier Trans- 
form (FFT) length and the number of FFT’s averaged to create the power spectrum, and 
(4) the variance also depends on the moment estimation algorithm used to extract the sig- 
nal power from the noisy power spectrum. In view of these difficulties, we have ap- 
proached the problem of estimating the number of bits heuristically, using the expression 
in Bringi and Chandrasekar (2001) to approximate the standard deviation of the mean 

power ( P ) of a sample of N P correlated pulses, <j[P(dB)] : 

<j[P(dB)\ = \OLog lo (l + -4=J Z a-^)ex P (-l^^)) 

v N P V=-(1V,-1) N P * 

Here, T S =PRF~ 1 is the pulse repetition interval and Ov is the standard deviation of the ve- 
locity. For a dwell time w=0.233 seconds, N P =Td*PRF=\280 samples. The result is 

<j\P{dB)]~ 0.12 dB for 2=0.0032 m and av=±2.5 m-sec" 1 . The standard deviation of mean 
power in dB is the same as the reflectivity factor Z e in (dBZ). To separate the effect of 
compression noise from the quantization noise, we introduced a noise threshold (Ts) that 

was set to 1% ofer[.P(<ifi)] , or 7^=0.0012 dB. Then, the number of bits is given by: 

b = Round (Logi(DRJT 8))=V1 ', where the dynamic range of the radar, DR= 80 dB. We 

used 18 bits to guard against the possibility of clipping. Also, the resulting integer data 
are considered as representative of the raw, integer receiver counts. 

There are two types of DWT to choose from: an integer DWT and a floating point 
DWT, to be noted as float DWT for brevity. Fully lossless compression can only be 
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206 achieved with the integer DWT while the float DWT generally provides higher perfor- 

207 mance in tunable (i.e., lossy) applications. After applying a 2-D wavelet transfonn to the 

208 data, the bit-plane encoder is employed for accurate compression rate control in the lossy 

209 mode. The CCSDS algorithm has demonstrated excellent perfonnance when applied to 

210 various types of images. However, the performance of the algorithm would degrade in 

211 the presence of large amounts of random noise. The CCSDS standard was chosen for 

212 evaluation for several reasons: first, ground-based radars can be considered as proxies for 

213 those employed in spacebome observation platforms; secondly, the standard was created 

214 to process space instrument data with onboard processing constraints that include limited 

215 processing power and memory, as well as other effects arising from the data packetiza- 

216 tion scheme, etc. Furthermore, radiation- tolerant hardware has already been developed 

217 (e.g., Winterrowd et al., 2011) and integrated into NASA’s mission, greatly reducing the 

218 risk and cost for future applications involving radar instruments. The results of this study 

219 therefore can serve as indicators of the expected levels of performance of data compres- 

220 sion for spaceborne radars attainable by this algorithm. 

221 

222 2.3 Data Products 

223 

224 The SMARTLabs/ACHIEVE radar provides measurements of the horizontal and 

225 vertical components of the reflectivity, moments of the hydrometeor velocity and the lin- 

226 ear depolarization ratio from which various meteorological products can be derived that 

227 characterize clouds and precipitation. This study focuses on the reflectivity field data 

228 product since it exhibits the largest variability and dynamic range, making it ideally suit- 

229 ed for evaluating the data compression algorithm. The approach taken here is to use the 

230 uncompressed and compressed reflectivity fields Z (mm 6 -m' 3 ), to derive: rainfall rate R 
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231 (mm-hr' 1 ), liquid water content W (g-m 3 ), and median particle size D 0 (cm). By compar- 

232 ing the results, we can investigate how non-linearities propagate error introduced by the 

233 compression/decompression process and affect the derived microphysical parameters in a 

234 way that is more insightful than merely subtracting the compressed and uncompressed 

235 reflectivity fields. To attain this objective it is necessary to introduce a set of working as- 

236 sumptions and to propose a model. Regarding the former, the analysis will be based on an 

237 input/output relationship X*=®*(Z{«,c}) where Xa is the derived field of interest (i.e., X=R, 

238 W, or Do), ®a is the non-linear function that accepts the uncompressed or compressed re- 

239 flectivity Z« or Z c respectively and k is a field identifying index that assumes \R, W, D } . 

240 Because interest is centered on investigating the effects of non-linearity, the function (Da 

241 can in principle be arbitrary. However, such arbitrariness can easily either grossly ampli- 

242 fy the compression error, or underrepresent its effects, hence the need of a physically- 

243 based model to introduce constraints. To model the electromagnetic scattering, we use 

244 the well-known fact that radar echoes from hydrometeors depend on the moments of the 

245 particle size distribution (PSD). Knowing, the PSD allows the derivation of other prod- 

246 ucts from the same PSD such as R, W, or Do. To this end, we referred to the PSD in the 

247 seminal work of Ulbrich (1981) and Rosenfeld and Ulbrich (2003) who made significant 

248 progress in addressing the longstanding question of the connections between raindrop- 

249 size distributions and radar reflectivity-rainfall rate (Z-R) relationships. 

250 The PSD we employed is the gamma distribution given as: 

251 N{D) = N 0 D»Exp{-KD) 0 < D < D max? (1) 

252 

253 where D is the equivolume spherical diameter of the particles and No, the number concen- 

254 tration (m^-cm 1 ). The slope parameter is designated by A (cm 1 ), and the shape parame- 
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255 ter, pi (dimensionless), is an exponent that can have positive or negative values. The di- 

256 ameter varies from zero to a maximum of Dmax. This PSD has been considered adequate 

257 (e.g., Ulbrich, 1981; Rosenfield and Ulbrich, 2003) in characterizing precipitation since it 

258 yields simple expressions for its moments in the limit of Dmax^cc. Table 1 illustrates how 

259 the rainfall rate (R) is related to the median particle size (Do) and the liquid water content 

260 ( W) via the reflectivity (Z) as given by Ulbrich (1981), derived from Eq. (1). 

261 The values of the parameters: No, pi, b, 8, re. A, s, and £ required by the formulae 

262 in Table 1, were compiled by Ulbrich (1981) who references 23 investigations extending 

263 from (1953-2002) that characterize precipitation ranging from stratiform to convective in 

264 the form of power-law Z-R relationships. The aforementioned parameters were inferred 

265 from S-, C- and X-band radar measurements and whose values define the model parame- 

266 ter space used in our analyses. In this study, W-band reflectivity data were used to calcu- 

267 late rainfall rate, liquid water content and median volume diameter fields. The rainfall 

268 rate from each model was first computed and then propagated to calculate the liquid wa- 

269 ter content and median volume diameter, according to Table 1. These fields were then 

270 compared to those calculated from the uncompressed data. The microwave frequencies 

271 employed by the authors cited by Ulbrich (1981) differ from the W-band. However, the 

272 analyses presented are nevertheless useful in evaluating the lossy compression algorithm, 

273 considering the uncertainties introduced by the non-uniqueness of the PSD and the largest 

274 measured amplitude of the W-band reflectivity. 

275 A search of the W-band data reveals the largest reflectivity to be 0.678 dBZ, a value 

276 that suggests the presence of light drizzle (<0.2 min-hr" 1 ), which is consistent also with 

277 our visual observations of the event. In the absence of rain or when light drizzle is pre- 
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278 sent, Rayleigh scattering by the cloud/water particles dominates as it does at the longer 

279 wavelengths used by the investigators referenced by Ulbrich (1981). The relatively small 

280 reflectivity is significant because otherwise, at the nominal frequency of 94 GHz, heavy 

281 precipitation characterized by large reflectivity would give rise to Mie scattering and be 

282 strongly attenuating, further exacerbating uncertainties in the interpretation of the rainfall 

283 rate. This contrasts with radars operating at longer wavelengths (e.g., S-, C- and X-band) 

284 where attenuation by heavy precipitation is significantly reduced. 

285 Finally, the W-band is not an atmospheric clear window, since water vapor and ox- 

286 ygen are actively absorbing gases in this region of the spectrum, the former dominating 

287 the latter. Thus, the reflectivity must normally be corrected for the 2-way attenuation by 

288 the absorbing gases and by cloud/precipitation particles. Using attenuation models given 

289 by Liebe (1993), the one-way attenuation rate on the day of the measurement, by water 

290 vapor and drizzle, at the surface was calculated to be 1.5 dB-km' 1 and decreasing with 

291 increasing altitude. As can be seen from Fig. lb, light drizzle extends to approximately 1 

292 km, thus eliminating the need for this correction. The uncertainties just described are 

293 much greater than those produced by compression noise, as will be seen. To summarize, 

294 the purpose of these analyses using 23 models of the PSD is to explore the impact of data 

295 compression noise inherent in the decompressed data on the meteorological fields previ- 

296 ously discussed and not in accurate retrievals of the parameters characterizing an as- 

297 sumed PSD. To carry out this objective we employed mathematical models characterized 

298 by frequently employed power-law non-linearities (e.g., Lohmeier et al., 1997; Uijlen- 

299 hoet, 2001) over a broad range of exponents, with particular interest in the amplification 

300 of error in R, W and Do. The pervasiveness of power-law relationships is evident in the 
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literature; these have even been developed to relate precipitation in the form of snowfall 
rate to radar reflectivity at W-band (e.g., Matrosov et al., 2008). 

3. Results 

After applying various degrees of lossy compression on the digital counts, the re- 
constructed reflectivity values were first compared to the original values directly from the 
radar. A root-mean-square error ( RMSE) criterion was employed to detennine the maxi- 
mum lossy compression in tenns of compression ratio (CR) corresponding to ±0.5 dB 
uncertainty in the radar reflectivity as it is commensurate with that introduced by the ra- 
dar calibration procedure. This statistic measures the difference between reflectivity val- 
ues compressed/decompressed by the CCSDS algorithm and the reflectivity values actu- 
ally observed. It can also be used as a measure of error in products that are derived from 
the compressed reflectivities as described below. Figures 3a-3d, computed by subtracting 
the compressed reflectivities from the uncompressed reflectivities, show the noise intro- 
duced by float DWT and integer DWT modes of compression for different values of CR. 
Such pixel differences are aggregated by the RMSE into a single, global measure of error 
attributed to compression noise introduced by the CCSDS algorithm. The RMSE of the 
compressed variable Zccsds is defined as the square root of the mean squared error: 


RMSE 




robs 

,j 


y CCSDS \ 2 

Aj ) 


NM 


( 2 ) 


where Z° bs is the observed reflectivity, Zf csm is the compressed reflectivity at the same 

pixel location (/, j) in the profile and NM, the number of pixel elements. The calculated 
RMSE values conveniently have the same units as the residuals. This error criterion, by 
virtue of the squaring process gives disproportionate weight to large errors by comparison 
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325 to either the mean absolute error (MAE) or mean error (ME) that employ the size of the 

326 residual, not its square. The ME statistic yields a signed measure of the error and is indic- 

327 ative of positive/negative bias. The MAE criterion yields results similar in magnitude, but 

328 smaller than the RMSE. The reason for employing the RMSE is that because it is sensitive 

329 to outliers, it can be used as a diagnostic to identify the location(s) in an image where the 

330 CCSDS introduces large compression noise errors and thus gain some insight as to what 

331 properties of the image cause undesirable algorithmic behavior. We shall describe and 

332 illustrate a simple, local measure of bias later in this section, where it can be associated 

333 with measurements taken at a particular time. We note here that since the compression 

334 algorithm introduces bias and variance, these components are combined in the mean 

335 squared error. 

336 From the RMSE curves shown in Fig. 4, it is seen that for the radar reflectivity dis- 

337 tortion range of ±0.5 dB, a data CR of 15 is achievable. As expected, the RMSE increases 

338 monotonically with increasing CR and the float DWT performs slightly better than the 

339 integer DWT. The effects of the different compression ratios on other radar data products 

340 are illustrated in Figs. 5a-5c. These figures were calculated by using Eq. (2) but with 

341 Z° b j and Zf ( j SDS now replaced by the meteorological fields of interest, derived from the 

342 unperturbed and perturbed reflectivities, respectively. In particular, for every product 

343 computed from the 23 different meteorological models, the product with the largest 

344 RMSE, i.e., the worst compression result, was selected for presentation. A second search 

345 was also performed to locate the minimum RMSE. The RMSE for the different products 

346 are not necessarily from the same models. In all cases, the RMSE errors increase with in- 

347 creasing CR and the results of using integer DWT are in excellent agreement with those 
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348 calculated using float DWT. Taking the logarithm of the RMSE permits visualizing the 

349 maximum and minimum curves on the same plot. 

350 Figures 6a and 6b addresses the question of the distribution of bias across the image 

351 at all acquisition times for CR=15. To calculate this local bias, a series of linear, least- 

352 squares regressions was performed at the different measurement times, between reflectiv- 

353 ity values in the original and compressed image. Thus, 7,709 independent regressions 

354 were calculated using 524 points per regression. In the absence of compression noise, the 

355 resulting line must have unity slope and zero intercept. The latter is the desired measure 

356 of bias. However, as can be seen in the histogram of Fig. 6a, the CCSDS algorithm intro- 

357 duces a bias (for lossy compression), whose largest value of 0.225 dB falls within the 

358 ±0.5 dB imposed requirement. Figure 6b illustrates the slope and intercept at different 

359 measurement times. For all regressions, the slope is nearly unity, suggesting that the 

360 CCSDS algorithm does not introduce nonlinearities, further attested by the fact that the 

361 minimum correlation coefficient found is 0.995 and by the observation that variances of 

362 the uncompressed and compressed reflectivities at all measurement times lie on a 45° line 

363 as shown in Fig. 6c. Analyses of the spread of points about this line, depicted by Fig. 6d 

364 indicate that the largest difference in the standard deviations between the compressed and 

365 uncompressed reflectivities is 0.44 dB which also falls within the ±0.5 dB uncertainty in 

366 calibration. The calculations used to produce Fig. 6d account for the correlations between 

367 the compressed and uncompressed reflectivities at all sampling times. 

368 The methods just described only produce convenient, two-point summary statistics 

369 and cannot provide infonnation about the shape of the error distribution. Shape infor- 

370 mation can be obtained from the error histogram, but plotting such figures for reflectivi- 
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371 ties at all observation times and for all derived products is impractical. However, global 

372 plots of errors are feasible as shown in Figs. 7a-7d. The figures display histograms of the 

373 differences in Z, R , W and Do between uncompressed and compressed data over the im- 

374 age, for the model that exhibits the largest RMSE in these fields at a CR value of 1 5 using 

375 float DWT. Biases are indicated by symmetric histograms not centered at the origin or by 

376 highly asymmetric histograms that include the origin. For example, it is seen that the re- 

377 flectivity exhibits a small bias, considering that out of a total of 4,039,516 points, about 

378 600,000 are without error and that the error mass pedestal is nearly symmetric. The bias 

379 is located slightly to the left of the origin and the dynamic range of uncertainty at the base 

380 of the histogram extends from -1 to +1 dBZ. It is also seen that R and W are relatively 

381 insensitive to compression noise in the reflectivity and do not exhibit undesirable bias 

382 since the error distribution is essentially symmetrical, centered about the origin. In loss- 

383 less compression, all the plots would be delta functions centered at the origin. Finally, the 

384 error distribution in the median volume diameter, Do exhibits a small asymmetry, with a 

385 mean of -2xl0" 5 mm and standard deviation of 2.97xl0' 4 mm, suggesting that it is more 

386 sensitive to compression noise than R or W. 

387 

388 4. Concluding Remarks and Future Work 

389 

390 From this preliminary study, it can be seen that a lossy compression ratio of at least 

391 1 5 can be achieved (depending on the meteorological situation) with an acceptable radar 

392 reflectivity noise margin of ±0.5 dB. For this value of the compression ratio, the derived 

393 products incur insignificant error. When rain rate, liquid water content and median vol- 

394 ume diameter fields are computed from the reflectivity data using 23 different models, 

395 the worst RMSE is below 10' 3 over the full range of tested compression ratios from 2 to 
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396 24. This is significant in that for a fully lossless compression, on the contrary, a compres- 

397 sion ratio of 1 .2 is observed instead. The implication is that no appreciable data reduction 

398 can be achieved if a fully lossless compression technique is employed, and such low 

399 compression is attributed to the inherent noisy characteristics of radar signals. As long as 

400 the compression technique introduces noise in the reflectivity that is below the noise 

401 margin set by the calibration, derived products dependent on the reflectivity will be neg- 

402 ligibly perturbed. Furthermore, the analyses presented have tacitly assumed that the radar 

403 calibration does not change during the observation period(s). Our study was performed 

404 on one set of data acquired in light drizzle and rain. To fully characterize the effects of 

405 compression on weather radar signals, extensive tests will be needed for data acquired 

406 under different weather conditions. The analyses of this dataset are not limited to reflec- 

407 tivity but can include polarimetric variables such as the linear depolarization ratio and the 

408 differential reflectivity. To further probe the effects of compression on meteorological 

409 products, tests will be conducted using a numerical retrieval technique to infer profiles of 

410 parameters that define the PSD in clouds and precipitation. The analyses presented have 

411 focused on a complex cloud system from which a compression ratio (i.e., 15) was de- 

412 rived. In the future, more comprehensive analyses will be performed for nominal terres- 

413 trial cloud systems; in turn, higher compression ratios can be expected. We note in clos- 

414 ing that lossy data compression has not yet been fully adopted by the remote sensing 

415 community. The current perception is that employing compressed images (or data) may 

416 ultimately affect the results of posterior processing (e.g., image classification and re- 

417 trieved products), potentially hindering the attainment of science goals. However, future 

418 satellite missions will certainly require the use of a suite of passive and active instru- 
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419 ments, raising the specter of bandwidth limitations and storage of unprecedented volumes 

420 of data. Thus, lossy compression may provide an effective means to mitigate these diffi- 

421 culties. Our approach to evaluating the effects of such compression, though preliminary, 

422 is insightful, providing a rational basis of addressing these issues. 

423 
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1. (a) Instrumentation setup of SMARTLabs- ACHIEVE (Aerosol-Cloud- 
Humidity Interaction Exploring & Validating Enterprise) mobile laboratory, 
shown a W-band cloud (94 GHz, pulsed) and X-band rain (10 GHz, FM-CW) 
radar mounted on a heavy-duty pedestal, with a zenith pointing K-band drizzle 
(24 GHz, FM-CW) radar and supplementary measurements by a ceilometer 
(910 nm, cloud base) mounted on the sidewall and an all-sky imager (cloud 
coverage), (b) an example of time series of W-band radar reflectivity collected 
on 8 May 2012 at NASA/GSFC, depicting drizzle and light-rain by a complex 
weather system passing overhead, (c) the corresponding linear depolarization 
ratio differentiating ice, melting, and water cloud phases, and (d) mean fall ve- 
locity indicating strongest rain occurred at ~7.5 minutes elapsed time. 

2. Optimal application of the CCSDS Image Data Compression standard onboard: 
CR can be determined for instrument i when compression is executed with in- 
put from Path 1; or the final CR can be assigned after rate optimization is per- 
formed on multiple instruments via the Path 2 input to the down link processor 
(DLP). Onboard solid-state recorder (SSR) holds the coded bit stream until 
down link is scheduled. Readjusting CR at DLP is simply achieved by truncat- 
ing the coded bit stream appropriately for each instrument. 

3. The distortion of radar reflectivity produced (a) using a CR value of 2 and float 
DWT compression mode; (b) as in (a), except for using integer DWT compres- 
sion mode; (c) as in (a), except for CR=24; and (d) as in (b), except for CR=24. 

4. The root-mean-square error as a function of CR under both float and integer 
DWT compression modes. 

5. Response of (a) rainfall, (b) liquid water content and (c) median volume diame- 
ter models to data compression noise produced by integer DWT and float DWT, 
illustrating the minimum and maximum root-mean-square errors. Both the up- 
per and lower pairs of curves are practically indistinguishable; this shows that 
these derived products are insensitive to the chosen mode of data compression. 
These errors are not necessarily computed from the same model. See text for de- 
tails. 

6. (a) Histogram of bias variations in the radar reflectivity image for CRM 5, (b) 
the scatter plot of their slopes at all observation times, calculated from a linear 
regression model for CRM 5 and float DWT, (c) plot of variance comparisons 
computed for compressed and uncompressed reflectivities at corresponding col- 
umns of the reflectivity images, and (d) plots of variance differences computed 
from the compressed and uncompressed reflectivity images at corresponding 
column locations and accounting for correlations. 

7. Global error distribution of (a) radar reflectivity, (b) rainfall rate, (c) liquid wa- 
ter content and (d) median volume diameter, computed for CRM 5. 
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533 TABLE 1. Relationships between radar reflectivity and rainfall rate, median particle diam- 

534 eter and liquid water content 
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Figure 1. (a) Instrumentation setup of SMARTLabs- ACHIEVE (Aerosol-Cloud- 
Humidity Interaction Exploring & Validating Enterprise) mobile laboratory, 
shown a W-band cloud (94 GHz, pulsed) and X-band rain (10 GHz, FM-CW) 
radar mounted on a heavy-duty pedestal, with a zenith pointing K-band drizzle 
(24 GHz, FM-CW) radar and supplementary measurements by a ceilometer 
(910 nm, cloud base) mounted on the sidewall and an all-sky imager (cloud 
coverage), (b) an example of time series of W-band radar reflectivity collected 
on 8 May 2012 at NASA/GSFC, depicting drizzle and light-rain by a complex 
weather system passing overhead, (c) the corresponding linear depolarization 
ratio differentiating ice, melting, and water cloud phases, and (d) mean fall ve- 
locity indicating strongest rain occurred at -7.5 minutes elapsed time. 
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Figure 2. Optimal application of the CCSDS Image Data Compression standard onboard: 
CR can be determined for instrument i when compression is executed with in- 
put from Path 1 ; or the final CR can be assigned after rate optimization is per- 
formed on multiple instruments via the Path 2 input to the down link processor 
(DLP). Onboard solid-state recorder (SSR) holds the coded bit stream until 
down link is scheduled. Readjusting CR at DLP is simply achieved by truncat- 
ing the coded bit stream appropriately for each instrument. 
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Figure 3. The distortion of radar reflectivity produced (a) using a CR value of 2 and float 
DWT compression mode; (b) as in (a), except for using integer DWT compres- 
sion mode; (c) as in (a), except for CR=24; and (d) as in (b), except for CR=24. 
A low gray-level resolution clearly highlights the aforementioned distortions. 
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Figure 4. The root-mean-square error as a function of CR under both float and integer 
DWT compression modes. 
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Figure 5. Response of (a) rainfall, (b) liquid water content and (c) median volume diame- 
ter models to data compression noise produced by integer DWT and float DWT, 
illustrating the minimum and maximum root-mean-square errors. Both the up- 
per and lower pairs of curves are practically indistinguishable; this shows that 
these derived products are insensitive to the chosen mode of data compression. 
These errors are not necessarily computed from the same model. See text for de- 
tails. 
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Figure 6. (a) Histogram of bias variations in the radar reflectivity for CR=15, (b) the scat- 
ter plot of their slopes at all observation times, calculated from a linear regres- 
sion model for CR=15 and float DWT, (c) plot of variance comparisons com- 
puted for compressed and uncompressed reflectivities at corresponding columns 
of the reflectivity images, and (d) plot of variance differences computed from 
the compressed and uncompressed reflectivity images at corresponding column 
locations and accounting for correlations. 
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595 Figure 7. Global error distribution of (a) radar reflectivity, (b) rainfall rate, (c) liquid wa- 

596 ter content and (d) median volume diameter, computed for CR=15. 
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